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INTRODUCTORY  GUIDE  TO  DECISION-AIDING  SOFTWARE 


L. 

1.0  INTRODUCTION 


“The  planning  and  conduct  of  war  have  acquired  a  precision, 
i  a  swiftness,  and  a  thoroughness  before  unknown.  The  proper 

solution  of  military  problems  requires  the  reaching  of  a  sound 
decision  as  to  what  is  to  be  done. " 


Sound  Military  Decisions 
U.S.  Naval  War  College.  1 936 


Despite  our  extraordinary  technological  advancement, 
reaching  a  sound  decision  to  solve  a  complex,  time-critical 
military  problem  is  as  formidable  a  task  today  as  it  was  in 
1936,  even  for  the  most  seasoned,  imaginative,  and  intelli¬ 
gent  decision  maker.  Indeed,  compelling  research  spanning 
the  forty-four  year  interval  suggests  that  holistic,  unaided 
decision  making  often  proves  faulty  when  applied  to  complex 
problems  involving  conflictive  objectives,  uncertainty,  and 
risk.^ 


Motivated  by  the  strength  of  that  research  evidence,  by 
the  critical  nature  of  today's  national  security  decisions, 
by  the  heavy  responsibilities  commonly  imposed  on  Defense 
decision  makers,  and  by  the  promise  of  innovative  technology 
to  aid  and  promote  accountability  in  the  decision-making 
process,  the  Defense  Advanced  Research  Projects  Agency 
(DARPA)  recently  sponsored  a  program  of  basic  research  and 


The  literature  is  vast.  Since  1970  more  than  one  thousand 
books,  articles,  and  technical  reports  have  been  published 
on  the  subject.  See,  for  example,  Slovic,  Paul;  Fischhoff, 
Baruch;  and  Lichtenstein,  Sarah,  Behavioral  Decision  Theory, 
Technical  Report  DDI-7  (Eugene,  Oregon:  Oregon  Research 
Institute,  September  1976). 
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exploratory  development  to  examine  decision-making  processes 
and  to  develop  practical  decision-aiding  methodology  for 
application  within  the  Department  of  Defense  (DoD) .  Spanning 
the  seven-year  period  from  1972  to  1979 ,  DARPA' s  Advanced 
Decision  Technology  Program  produced  incontroversial  evidence 
that  underscored  the  need  for  practical  and  useful  decision- 
aiding  methodology  for  Defense  decision  makers.  In  response 
to  that  need  and  as  an  integral  part  of  the  program,  DARPA 
sponsored  the  development  of  decision-aiding  computer  soft¬ 
ware  that  embodies  the  fundamental  concepts  of  the  management 
discipline  of  decision  analysis. 

Most  of  the  software  decision  aids  are  generic  model¬ 
building  aids.  That  is,  in  most  cases  the  software  was 
designed  to  enable  a  user  knowledgeable  in  decision  analysis 
methodology  to  build  and  tailor  a  model  to  fit  a  specific 
decision  problem  under  consideration.  Thus,  any  one  decision 
aid  may  be  used  to  create  and  exercise  any  number  of  specific 
decision-aiding  models.  The  resulting  models  may  be  con¬ 
structed,  stored,  retrieved,  edited,  extended,  and  exercised 
electronically  using  computer  technology. 

The  family  of  decision-aiding  software  was  applied  ex¬ 
tensively  on  an  exploratory  basis  to  the  solution  of  diverse, 
important,  real-world  problems  throughout  the  DoD  during  the 
period  1976  to  1979.  The  pilot  applications  of  the  decision 
aids  included,  but  were  not  limited  to: 

a.  use  by  the  staff  sections  of  Headquarters,  U.S. 
European  Command  and  its  three  component  service 
commands— the  aids  were  applied  to  more  than  forty 
specific  decision  problems  and  remain  in  active 
use  at  the  present  time; 
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b.  use  by  the  U.S.  Army  in  the  evaluation  of  alterna¬ 
tive  single  channel  ground  and  airborne  radio 
system  (SINCGARS)  configurations  for  the  post-1980 
time  period; 

c.  use  by  the  Intelligence  Center ,  Pacific  in  assess¬ 
ing  the  likelihood  of  adversary  actions; 

d.  use  by  the  National  Security  Council  in  deciding 
what  kinds  of  computer  technology  were  appropriate 
for  export  sale  to  the  Soviet  Union; 

e.  use  by  the  Central  Intelligence  Agency  and  DoD/ 
International  Security  Affairs  (ISA)  in  applying 
Pareto-optimal  negotiation  strategies  to  the 
Panama  Canal  treaty  negotiations  and  to  the 
Philippine  base-rights  treaty  negotiations; 

f.  use  by  the  Department  of  Transportation  in  con¬ 
ducting  international  negotiations  on  tanker 
safety; 

g.  use  by  Headquarters,  U.S.  Marine  Corps  as  an  orga¬ 
nizing  vehicle  for  the  preparation  of  the  FY  1979 
and  FY  1980  Program  Objectives  Memorandum  (POM) ; 
and 

h.  adoption  of  the  methodology  by  the  U.S.  Marine 
Corps  as  the  basis  for  a  standard  procedure  used 
to  assess  the  readiness  of  combat  units. 

Those  and  many  other  diverse  pilot  applications  of  the 
decision  aids  over  the  three-year  period  successfully  demon¬ 
strated  their  potential  applicability  to  important  Defense 
decision  problems. 
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The  documentation  effort  described  herein  supports 
DARPA's  desire  to  transfer  the  decision-aiding  software  to 
the  broadest  possible  user  population.  However ,  since  few 
potential  users  of  the  aids  have  access  to  the  IBM  5100 
: computer  or  to  the  APL  computer  programming  language  on 
which  the  aids  were  developed,  the  documentation  is  itself 
generic  in  nature.  That  is,  it  has  been  written  indepen¬ 
dently  of  computer  and  programming  language.  Thus,  a  pro¬ 
spective  user  agency  should  be  able  to  apply  the  documents 
to  the  development  of  software  that  implements  an  opera¬ 
tional  version  of  the  aids  on  any  host  computer  using  any 
computer  language. 

This  manual  briefly  discusses  the  need  for  decision 
aids,  identifies  the  seven  decision  aids  that  were  chosen 
for  documentation,  and  describes  the  nature  of  the  documents 
available  to  prospective  users.  A  complete  list  of  the 
documents  appears  in  Section  10.0. 


2.0  DECISION  AIDS 


“The  responsibility  for  making  a  decision  is  solely  that  of  the 
commander,  and  the  precise  manta! processes  he  uses  in  its  formulation 
are  his  own  concern.  Regardless  of  the  techniques  employed,  a  sound 
decision  will  reflect  a  thorough  analysis  of  all  information  pertinent  to 
the  solution. " 


FMFM  3-1 .  Command  and  Staff  Action 
U.S.  Marine  Corps,  1970 


2.1  The  Need  for  Decision  Aids 

More  than  twenty  years  ago  Herbert  Simon,  the  1978 
Nobel  Laureate,  wrote  that  the  "capacity  of  the  human  mind 
for  formulating  and  solving  complex  problems  is  very  small 
compared  with  the  size  of  the  problems  whose  solution  is 
required  for  objectively  rational  behavior  in  the  real 
world — or  even  for  a  reasonable  approximation  to  such  objec¬ 
tive  rationality."3-  Rational  decision  making  is  even  fur¬ 
ther  strained  in  crisis  decision  situations,  which  are 
normally  attended  by  urgency,  risk,  conflictive  objectives, 
inconclusive  information,  and  unclear  personal  judgments. 
Understandably,  complex  crisis  situations  offer  all  too 
abundant  opportunities  for  misperception,  miscommunication, 
and  misunderstanding.  It  iB  rare  that  a  decision  choice 
reflects  the  implications  of  a  thorough  analysis  of  all  the 
information  pertinent  to  the  solution. 


^Herbert  A.  Simon,  Models  of  Man:  Social  and  National 
(New  York:  Wiley,  155)) .  ' 


5 


Indeed #  one  of  the  major  results  of  decision  research 
over  the  past  decade  is  that  decision  makers  and  their 
staffs  systematically  and  predictably  violate  the  principles 
of  sound  decision  making  when  they  attempt  to  cope  with  the 
iincertainty  that  attends  most  decision  problems.  Humans  are 
notoriously  biased  when  judging  the  likelihood  of  future 
events  or  otherwise  performing  probabilistic  tasks.  Fur¬ 
thermore#  the  research  shows  that  the  deficient  reasoning 
that  humans  apply  to  probabilistic  tasks  is  present  in  the 
problem  structuring  and  value  assessment  tasks  as  well.  The 
biased  reasoning  can  be  traced  to  various  intuitive  and 
superficial  strain-reducing  strategies  that  decision  makers 
commonly  employ  to  analyze  information. 

The  deficiencies  of  those  strategies  manifest  them¬ 
selves  in  several  areas  of  the  decision-making  process.  To 
illustrate  those  areas  consider  Figure  2-1#  which  is  a 
decision  tree  model  of  a  relatively  simple  decision  situa¬ 
tion.  The  model  depicts  a  decision  problem  having  two 
alternative  courses  of  action:  and  D2>  The  problem  is 

to  choose  one  of  those  two  alternatives.  Presumably#  the 
choice  is  irrevocable;  characteristic  of  most  important 
decisions#  once  the  choice  is  implemented  there  is  no  turning 
back. 

The  model  shows  that  if  course  of  action  is  selected# 
then  the  eventual  outcome  will  be  0^#  with  certainty.  How¬ 
ever#  should  course  of  action  D2  be  selected#  then  the 
eventual  outcome  will  be  either  02  or  0^#  depending  on  how 
the  uncertain  event  unfolds.  Presumably  the  key  uncertainty 
is  an  event  (such  as  weather  or  adversary  intentions)  not 
under  the  control  of  the  decision  maker. 

One  area  of  deficiency  is  that  of  coping  with  the 
uncertain  event.  Most  decision  makers  will  admit  to  the 


KEY  EVENTUAL 

DECISION  DECISION  UNCERTAIN  EVENT  DECISION 

HOCK  AiTENNATIVES  EVENT  OUTCOMES  OUTCOMES 


Figure  2-1 

A  SIMPLE  DECISION  TREE  MODEL 


uncertainty  they  face/  but  they  most  often  deal  with  it  by 
attempting  to  remove  the  uncertainty  altogether  through  a 
process  of  information  collection.  Instead,  they  should 
accept  the  probabilistic  nature  of  the  real-world  and  deal 
with  the  uncertainty  in  probabilistic  terms.  That  is, 
referring  to  Figure  2-1,  they  should  assess  the  relative 
likelihoods  of  the  two  event  outcomes,  E^  and  E2*  As  evi¬ 
dence  becomes  available,  it  should  be  processed  to  update 
those  relative  likelihoods.  Probability  theory  is  the 
relevant  methodological  base  for  assessing  and  communicating 
uncertainty. 

Another  distinct  area  of  deficiency  is  that  of  evalu¬ 
ating  the  relative  attractiveness  of  the  three  possible 
outcomes  of  the  decision  problem,  0^,  02,  and  Oj.  Sweeping 
absolute  judgments  (possibly  entangled  with  the  uncertain 
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•vent)  are  often  applied  to  that  task.  Instead,  the  decision 
maker  should  isolate  the  analysis  of  consequences  from  the 
analysis  of  uncertainty,  and  develop  criteria  for  assessing 
•relative  preferences  regarding  the  decision  outcomes. 

‘Utility  theory  is  the  relevant  methodological  base  for 
assessing  and  communicating  preference. 

Another  area  for  concern  is  that  of  logically  inte- 
grating  the  informed  probability  assessment  of  the  uncertain 
event  with  the  informed  utility  assessment  of  the  decision 
outcomes  to  arrive  at  a  final  choice:  to  implement  course 
of  action  or  D2<  The  assessment  and  integration  process 
is  a  formidable  task,  and  one  that  if  not  done  systematically 
often  leads  to  a  decision  choice  that  does  not  cohere  with 
the  organizational  value  structure. 

The  decision-aiding  software  described  in  this  Guide 
was  designed  to  promote  sound  decision  making  through 
systematic  normative  procedures  that  help  decision  makers 
overcome  the  judgmental  deficiencies  described  above. 

2.2  Decision  Analysis 

The  software  described  herein  fulfills  the  role  of 
methodological  decision  aids.  All  of  the  software  has  its 
roots  in  the  management  discipline  of  decision  analysis,  a 
decision-making  strategy  that  dates  from  the  mid-1960's  but 
whose  roots  extend  into  the  18th  century  concepts  of  proba¬ 
bility  and  utility.  Indeed,  probability  theory  and  utility 

2 

theory  together  form  the  foundations  of  decision  analysis. 


y  . .  "  -  ■ 

There  is  a  vast  literature  on  decision  analysis.  For  an 
introductory  treatment  the  reader  should  refer  to  Howard 
Raiffa,  Decision  Analysis  (Reading,  Massachusetts :  Addison 
Wesley,  1^68);  Dennisv.  Lindley,  Making  Decisions  (London: 
Wiley,  1971) t  Rex  V.  Brown,  Andrew  &.  Kahr,  and  Cameron  R. 
Peterson,  Decision  Analysis  for  the  Manager  (New  York:  Holt, 


Simply  put,  the  discipline  assists  planners  and  deci¬ 
sion  makers  in  choosing  between  alternative  courses  of 
action  by  systematically  decomposing,  quantifying,  and 
examining  the  implications  of  the  relevant  considerations, 
-however  subjective  and  tenuous  they  may  be,  that  bear  on  the 
decision  problem.  The  overall  goal  of  decision  analysis  is 
to  ensure  that  the  ultimate  plan  or  decision  choice  is  a 
fully  coherent  one;  that  is,  a  choice  that  is  fully  con¬ 
sistent  with  the  organizational  goals  and  objectives,  value 
structure,  and  informed  beliefs.  In  addition,  the  inherently 
quantitative  framework  imposed  by  decision  analysis  serves 
the  planning  and  decision-making  process  in  several  other 
ways.  For  example,  the  approach  clarifies  and  makes  explicit 
the  important  subjective  value  structure  and  rationale 
underlying  the  problem.  That  process,  in  turn,  builds 
additional  insight  into  the  problem,  promotes  accountability 
in  the  decision-making  process,  and  facilitates  communication 
and  understanding  among  all  of  those  involved  in  the  process. 

2.3  The  Seven  Decision  Aids 

Most  of  the  decision-aiding  software  is  generic  model¬ 
building  software  that  is  applicable  to  broad  classes  of 
decision  problems.  That  is,  as  opposed  to  providing  fixed, 
uniquely  configured  models  for  specific  applications,  most 
of  the  software  permits  the  user  to  construct  and  tailor  an 
individual  model  to  fit  the  particular  problem  at  hand. 
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Rinehart  and  Winston,  1974);  or  Scott  Barclay,  Rsx  V.  Brown, 
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and  Judith  Selvidge,  Handbook  for  Decision  Analysis,  Tech¬ 
nical  Report  TR-77-6-56  (McLean,  Virginia:  Decisions  and 
Designs,  Inc.,  1977). 
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The  aids  are  dascribad  briafly  below.  A  more  detailed 
description  of  each  aid  appears  in  the  following  seven 
t  chapters . 

t 

t  1.  DECISION  (Decision  Tree  Model)  -  This  generic 

model-building  software  aid  enables  users  to  con¬ 
struct,  specify,  and  exercise  classical  decision 
tree  models  of  the  kind  shown  in  Figure  2-1. 
DECISION  processes  the  specified  probabilities  of 
the  uncertain  events  and  the  assessed  utilities  of 
the  possible  decision  outcomes  to  arrive  at  a 
recommended  course  of  action. 

DECISION  is  described  in  Section  3.0. 

2.  OPINT  (Operations- Intelligence)  -  This  generic 
model- building  software  aid  enables  users  to  con¬ 
struct  and  specify  a  restricted  version  of  a 
decision  tree  model.  The  model  is  restricted  to 
one  decision  block  followed  by  one  key  uncertainty 
that  attaches  to  each  alternative  course  of 
action.  OPINT  assists  the  user  in  specifying  the 
consequences  of  the  decision  outcomes.  It  pro¬ 
cesses  the  specified  event  likelihoods  and  the 
assessed  values  of  the  decision  outcomes  to  arrive 
at  a  recommended  course  of  action. 

OPINT  is  described  in  Section  4.0. 

3.  INFER  (Inference)  -  This  generic  model-building 

, '  software  aid  enables  users  to  construct  and  spe¬ 

cify  influence  diagram  models  that  represent 
dependency  relationships  between  uncertain  future 
events.  INFER  processes  direct  and  conditional 
probability  assessments  to  produce  unconditional 


probabilities  for  any  event  of  interest.  INFER 
supports  Bayesian  updating  of  prior  probabilities 
in  light  of  new  evidence. 

INFER  is  described  in  Section  5.0. 

4.  HIER  (Hierarchical  Inference)  -  This  generic  soft¬ 
ware  aid  enables  a  user  to  assess  the  implications 
of  evidence  relating  to  hypotheses  of  interest  con¬ 
cerning  a  key  future  event. 

HIER  is  described  in  Section  6.0. 

5.  SCORING  RULE  -  This  specialized  software  aid  adminis¬ 
ters  a  subjective  probability  assessment  test  designed 
to  improve  the  calibration  of  probability  assessors. 

SCORING  RULE  is  described  in  Section  7.0. 

6.  EVAL  (Evaluation)  -  This  generic  model-building 
software  aid  enables  users  to  construct  and  specify 
multi -attribute  utility  models  for  evaluating  vari¬ 
ous  alternative  systems,  plans,  or  courses  of  action 
under  conditions  of  relative  certainty. 

EVAL  is  described  in  Section  8.0. 

7.  RAM  (Resource  Allocation  Model)  -  This  generic 
model-building  software  was  developed  specifically 
to  aid  those  managers  who  are  responsible  for  the 
annual  preparation  and  submission  of  the  Program 
Objectives  Memorandum  (POM) .  RAM  permits  users  to 
perform  cost/benefit  comparisons  and  trade-offs 
among  programs  competing  for  limited  funds. 

RAM  is  described  in  Section  9.0. 


3.0  DECISION 


DECISION,  the  name  of  the  system  described  in  this  sec¬ 
tion,  is  derived  from  Decision  Tree.  The  system  aids  the 
decision-making  process  by  enabling  users  to  construct,  spe¬ 
cify,  and  solve  classical  decision  tree  models  of  problems 
of  choice. 

3.1  Purpose 

The  overall  purpose  of  DECISION  is  to  assist  decision 
makers  in  building  and  exercising  decision  tree  models  that 
approximate  real-world  decision  problems.  A  typical  (but 
incomplete)  decision  tree  structure  that  could  be  imple¬ 
mented  using  DECISION  is  shown  in  Figure  3-1.  The  structure 
shows,  proceeding  from  left  to  right,  three  immediate  deci¬ 
sion  choices,  each  followed  by  uncertain  future  events  and/ 
or  subsequent  decisions  that  lead  eventually  to  the  possible 
outcomes.  The  user  must  specify  the  relative  likelihood 
(probability)  of  the  uncertain  events  and  the  relative 
desirability  (utility)  of  the  possible  outcomes. 

Consistent  with  the  user's  specifications,  DECISION 
will  indicate  the  rational  decision  choice  (the  one  having 
the  maximum  expected  utility) . 

3.2  Intended  Users 


Decision  makers  and  their  operations  and/or  planning 
staff.  Anyone  who  must  resolve  alternative  courses  of 
action  conditioned  by  multiple  objectives  and  key  uncertain¬ 
ties. 
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Figure  3-1 

A  TYPICAL  DECISION  TREE 


3.3  Potential  Areas  of  Application 

o  Tactical  operations 
o  Crisis  decision  making 
o  Contingency  planning 
o  Strategic  planning 

3.4  Technical  Concepts 

o  Decision  tree  modeling 

o  Probability  assessment 

o  Utility  assessment 
o  Multi-attribute  decision  theory 
o  Expected  value  theory 

3.5  System  Summary 

The  DECISION  software  permits  decision  makers  to  con¬ 
struct,  store,  retrieve,  and  revise  decision  tree  models  of 
the  kind  shown  in  Figure  3-1.  The  user  may  identify  multiple 
criteria  for  assessing  the  relative  preferences  (utility) 
regarding  the  various  decision  outcomes.  In  addition,  the 
user  must  specify  the  relative  importance  of  the  criteria. 

The  user  must  also  specify  the  probabilities  of  the  uncer¬ 
tain  events. 

DECISION  processes  the  user  specifications  to  produce 
the  values  of  the  expected  utility  for  all  of  the  branches 
of  the  tree.  The  overall  result  is  the  expected  utilities 
associated  with  the  initial  decision  branches. 

3.6  User  Inputs 

The  user  must  specify  (1)  the  format  of  the  decision 
tree,  (2)  the  criteria  to  be  used  for  evaluating  the  eventual 
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outcome  of  the  problem,  (3)  the  relative  Importance  of  the 
criteria,  (4)  the  relative  utility  of  the  possible  outcomes 
of  the  problem  for  each  ctiterion,  and  (5)  the  probability 
of  occurrence  of  each  uncertain  event. 

* 

3.7  System  Outputs 

DECISION  processes  the  input  data  to  produce  the  ex¬ 
pected  utility  associated  with  each  branch  of  the  decision 
tree.  The  major  result  of  interest  is  a  display  of  the 
utilities  associated  with  the  initial  decision  choices.  The 
rational  choice  is  that  course  of  action  having  the  greatest 
expected  utility. 

* 

Figure  3-2  shows  a  display  of  the  overall  results  that 
might  obtain  from  the  decision  tree  shown  in  Figure  3-1. 

A 

1  OVERALL  RESULT 


CRITERIA: 

CRIT.  WEIGHTS: 

NAT-S 

50 

DOM-A 

25 

FOR-A 

25 

TOTAL 

1)  RAID 

80 

42 

51 

63 

2)  WARN 

62 

74 

81 

70 

3)  AGENT 

41 

60 

72 

54 

^  Figure  3-2 

TYPICAL  RESULTS  MATRIX 

a 

The  figure  indicates  that  three  criteria  (the  impact  on 
national  security,  domestic  affairs,  and  foreign  affairs) 
were  used  to  evaluate  the  outcomes,  and  they  were  weighted 
50%,  25%,  and  25%,  respectively.  The  matrix  also  shows  the 
relative  satisfaction  each  course  of  action  provides  under 
each  criterion.  For  example,  having  chosen  the  RAID  course 
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of  action,  the  decision  maker  expects  80%  satisfaction  with 
respect  to  national  security,  42%  satisfaction  with  respect 
to  domestic  affairs,  and  51%  for  foreign  affairs,  leading  to 
a  weighted  overall  expected  utility  of  €3%.  Consistent  with 
■the  overall  results,  the  rational  choice  would  be  to  WARN, 
since  that  course  of  action  provides  the  greatest  expected 
utility. 


4 . 0  OPINT 


J 


OPINT,  the  name  of  the  system  described  in  this  sec- 
-tion,  is  an  abbreviation  for  Operations  and  Intelligence, 
reflecting  the  system's  applicability  to  operational  deci¬ 
sion  making  based  on  intelligence  estimates.  OPINT  permits 
the  user  to  implement,  under  a  severe  time  constraint,  a 
specialized  and  restricted  decision  tree  model  of  a  real- 
world  crisis  situation. 

4.1  Purpose 

OPINT  assists  crisis  decision  makers  in  rapidly  resolv¬ 
ing  a  difficult  problem  of  choice  in  the  face  of  one  key 
uncertainty  and  multiple  conflictive  objectives.  Based  on 
user  inputs,  the  system  identifies  the  course  of  action 
having  the  least  expected  regret. 

4.2  Intended  Users 

Crisis  decision  makers  and  their  operations  staff. 

4.3  Potential  Areas  of  Application 

o  Crisis  decision  making 
o  Contingency  planning 

4.4  Technical  Concepts  Underlying  the  Aid 

o  Decision  tree  modeling 
o  Probability  assessment 

o  Utility  and  regret  assessment 

o  Multi-attribute  utility  assessment 
o  Expected  value  theory 
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4.5  System  Summary 

OPINT  accommodates  a  highly  restricted  decision  tree 
format  that  accommodates  just  one  decision  and  one  key 
Uncertain  event.  The  uncertain  event  attaches  to  each 
decision  alternative.  Figure  4-1  shows  the  typical  format 
of  an  OPINT  decision  model.  Multiple  criteria  may  be  used 
to  specify  the  relative  dissatisfaction  (regret)  associated 
with  the  decision  outcomes. 

The  user  must  also  specify  the  relative  importance 
weights  of  the  criteria,  the  relative  likelihood  (proba¬ 
bility)  of  the  uncertain  event  outcomes,  and  for  each  cri¬ 
terion  the  relative  regret  associated  with  each  decision 
outcome . 


DECISION  DECISION 
BLOCK  ALTERNATIVES 


KEY 

UNCERTAIN  EVENT  DECISION 

EVENT  OUTCOMES  OUTCOMES 


Figure  4-1 

AN  OPINT  MODEL 
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OPINT  processes  the  input  information  to  produce  the 
expected  regret  associated  with  each  course  of  action.  It 
also  provides  a  sensitivity  analysis  function  that  displays 
the  optimal  choice  of  decision  alternatives  as  a  function 
‘of  changes  in  the  event  outcome  probabilities  and/or  changes 
in  the  weights  assigned  to  the  criteria. 

4.6  User  Inputs 

The  user  must  specify  (1)  the  courses  of  action  under 
consideration,  (2)  the  possible  outcomes  of  the  key  uncer¬ 
tainty  and  their  probabilities  of  occurrence,  (3)  the  cri¬ 
teria  to  be  used  in  judging  the  relative  dissatisfaction 
associated  with  the  decision  outcomes,  (4)  the  relative 
importance  of  the  criteria,  and  (5)  the  relative  regrets 
ssociated  with  the  decision  outcomes  for  each  criterion. 

4.7  System  Outputs 

OPINT  processes  the  input  information  to  produce  the 
expected  regret  associated  with  each  course  of  action.  The 
rational  choice  is  that  course  of  action  leading  to  the 
least  expected  regret. 

OPINT  also  produces  two  kinds  of  sensitivity  analysis. 
Figure  4-2  shows  the  sensitivity  of  the  courses  of  action  to 
changes  in  an  event  outcome  probability.  The  matrix  dis¬ 
plays  the  expected  regret  associated  with  each  of  the  four 
courses  of  action  (listed  vertically)  as  a  function  of 
changes  in  the  probability  of  the  event  NONE,  varied  from  0 
to  100%  in  steps  of  10%.  For  each  probability,  the  least 
expected  regret  is  identified  by  an  asterisk.  Arrows  indi¬ 
cate  a  change  in  the  best  course  of  action  due  to  the  change 
in  probability. 
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EXPECTED  VALUE  WHEN 
PROBABILITY  OF  NONE  IS: 


0 

10 

20 

30 

40 

50 

60 

70 

80  . 

90 

100 

* NORMAL  -57 

-51 

-45 

-40 

-34 

-28 

-23* 

-17* 

-11* 

-  6* 

0* 

-LOW  PROF- 50 

-46 

-42 

-38 

-34 

-29 

-25 

-21 

-17 

-13 

-9 

MED  PROF-27 

-27 

-27 

-26* 

-26* 

-25* 

-25 

-25 

-24 

-24 

-24 

EVAC  PST-26* 

-26* 

-26*  -27 

-27 

-28 

-28 

-28 

-29 

-29 

-29 

+  + 


Figure  4-2 

SENSITIVITY  TO  PROBABILITY 


Figure  4-3  shows  the  sensitivity  of  the  courses  of 
action  to  changes  in  the  importance  weight  of  a  specified 
criterion.  The  format  is  identical  to  that  of  Figure  4-2, 
except  that  the  criterion  importance  weight  is  varied  from 
0%  to  100%  in  lieu  of  the  event  outcome  probability. 


EXPECTED  VALUE  WHEN 
WEIGHT  OF  EXPOSURE  RISK  IS: 


0 

10 

20 

30 

40 

50 

60 

70 

80 

90 

100 

NORMAL  -22* 

-23* 

-25* 

-26* 

-27 

-29 

-30 

-31 

-33 

-34 

-35 

LOW  PROF- 3 2 

-31 

-31 

-30 

-29 

-29 

-28 

-27 

-27 

-26 

-25 

MED  PROF-44 

-39 

-35 

-31 

-27* 

-23* 

-19* 

-15* 

-11 

-7 

-3 

EVAC  PST- 4 9 

-44 

-40 

-35 

-30 

-25 

-20 

-15 

-10* 

-5 

-1 

t  + 


Figure  4-3 

SENSITIVITY  TO  CRITERION  WEIGHT 
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5.0  INFER 


INFER,  the  name  of  the  system  described  in  this,  sec¬ 
tion,  is  an  abbreviation  for  Inference,  reflecting  the 
logical  process  implemented  by  the  software. 

5.1  Purpose 

The  purpose  of  the  INFER  system  is  to  assist  intelli¬ 
gence  analysts  and  forecasters  by  providing  them  a  systematic 
procedure  for  organizing  and  analyzing  difficult  probability 
assessments  by  using  inference  models  and  influence  diagrams. 

5.2  Intended  Users 

Intelligence  analysts,  planners,  and  forecasters. 

5.3  Potential  Areas  of  Application 

o  Preparation  of  intelligence  estimates 

o  Crisis  decision  making 

o  Contingency  planning 

5.4  Technical  Concepts  Underlying  the  Aid 


o  Influence  diagrams 

o  Direct  probability  assessment 

o  Conditional  probability  assessment 
o  Bayesian  inference 


5.5  System  Summary 

INFER  aids  the  probability  assessment  task  by  enabling 
users  to  create,  store,  retrieve,  exercise,  and  refine 


inference  models  that  approximate  causal  relationships 
between  several  uncertain  events. 

The  models  captured  by  INFER  are  based  on  a  methodo¬ 
logical  approach  known  as  influence  diagramming.  Figure  5-1 
shows  a  typical  influence  diagram.  The  diagram  indicates 
that  the  outcome  of  the  event  of  interest ,  Event  E,  is 
influenced  by  three  preceding  events.  Events  A,  B,  and  c. 

The  diagram  also  indicates  the  presence  of  an  indicator, 
Indicator  X,  whose  future  occurrence  will  modify  the  likeli¬ 
hoods  of  the  outcomes  of  Event  C  and  hence  Event  E. 

INFER  processes  the  user-specified  influence  diagram 
and  associated  assessments  to  produce  the  unconditional 
probabilities  of  the  outcomes  of  all  of  the  conditioned 
events.  The  major  result  is  the  probabilities  of  the  out¬ 
comes  of  the  key  event  of  interest  (Event  E  in  Figure  5-1) . 


5.6  User  Inputs 

The  user  must  specify  (1)  the  f onset  of  the  influence 
■diagram,  (2)  the  possible  outcomes  of  each  event,  (3)  proba¬ 
bilities  of  the  outcomes  of  each  unconditioned  event  (Event  A 
in  Figure  5-1) ,  (4)  conditional  probabilities  of  the  outcomes 
of  each  conditioned  event,  and  (5)  Bayesian  likelihood 
ratios  for  each  indicator. 

5.7  System  Outputs 

INFER  processes  the  user  input  information  to  produce 
the  unconditional  (marginal)  probabilities  of  the  outcomes 
of  each  conditioned  event.  Figure  5-2  shows  a  typical 
output  display.  The  diagram  in  Figure  5-2  indicates  that 
the  event  of  interest,  S.  MISSILES,  is  influenced  by  two 
other  events,  S.  INTENT  and  SUB  VISIT.  The  matrix  lists 
vertically  the  four  possible  outcomes  of  the  two  condition¬ 
ing  events  and,  in  parentheses,  their  joint  probabilities. 


|  S .  MISSILES  1 


I - ' - 1 

S.  INTENT  .SUB  VISIT 

- 1  I - 1  —  S.  MISSILES  — 


S.  INTENT/SUB  VISIT 

PRESENT 

NOT 

SUPPORT  R./YES 

(33) 

95 

5 

SUPPORT  R./NO 

(  8) 

1 

99 

NO  SUPPORT  AES 

(48) 

20 

80 

NO  SUPPORT/NO 

(11) 

1 

99 

MARGINAL  PROBABILITY 

32 

68 

Figure  5-2 
AN  OUTPUT  DISPLAY 
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It  also  shows  the  user-supplied  probabilities  for  the  two 
outcomes  of  S.  MISSILES  (PRESENT  and  NOT)  conditional  on  the 
joint  influencing  event  being  true.  The  result  of  interest 
is  the  unconditional  (marginal)  probabilities.  The  figure 
‘indicates  a  32%  probability  that  S.  MISSILES  are  present. 

The  user  may  set  indicators  either  ON  (observed)  or  OFF 
(not  observed) .  If  indicators  are  observed,  INFER  recomputes 
the  entire  chain  of  posterior  probabilities  based  on  the 
prior  probabilities  and  the  Bayesian  likelihood  ratios 
supplied  by  the  user. 


6.0  HIER 


HIER,  the  name  of  the  system  described  in  this  section, 
Is  an  abbreviation  for  Hierarchical  Inference,  reflecting 
the  logical  process  implemented  by  the  software. 

6.1  Purpose 

The  purpose  of  the  HIER  system  is  to  assist  intelli¬ 
gence  analysts  and  forecasters  by  providing  them  with  a 
systematic  procedure  for  organizing  and  analyzing  proba¬ 
bility  assessments  by  using  hierarchical  inference  models. 

6.2  Intended  Users 

Intelligence  analysts,  planners,  and  forecasters. 

6.3  Potential  Areas  of  Application 

o  Preparation  of  intelligence  estimates 

o  Crisis  decision  making 

o  Contingency  planning 

6.4  Technical  Concepts  Underlying  the  Aid 

o  Hierarchical  inference 

o  Conditional  probability  assessment 

o  Bayesian  inference 

6.5  System  Summary 

HIER  aids  the  probability  assessment  task  by  enabling 
users  to  construct  and  exercise  hierarchical  inference 
models  that  approximate  complex  causal  relationships  con¬ 
cerning  specific  hypotheses  about  an  uncertain  event.  HIER 


links  the  hypotheses  with  various  activities,  indicators, 
and  supporting  data  whose  occurrence  or  lack  of  occurrence 
would  influence  the  likelihood  of  each  hypothesis. 

*  Figure  6-1  shows  an  illustrative  hierarchical  struc¬ 
ture.  The  figure  indicates  that  the  relative  likelihood  of 
certain  hypotheses  (H)  would  be  directly  influenced  by 

certain  data  (D*  and  D^)  and  the  presence  of  certain  activi- 
2  4  5 

ties  (A  ,  A  ,  and  A  ).  Similarly,  the  activities  are  influ¬ 
enced  by  a  complex  structure  of  indicators  (I)  and  data  (D) . 
Note  that  the  input  (bottom- level)  nodes  are  all  data  nodes. 


Figure  6- 1 

ILLUSTRATIVE  HIERARCHICAL  STRUCTURE 


HIER  enables  the  user  to  construct  the  complete  hier¬ 
archical  structure  and  specify  the  various  conditional 
probabilities  and  direct  assessments  that  link  the  lower- 
level  data  nodes  of  the  structure  with  the  higher-level 
nodes.  HIER  processes  the  user  specifications  to  produce 
the  updated  relative  probability  for  each  hypothesis  about 
the  event  of  interest. 
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6.6  User  Inputs 

The  user  must  specify  (1)  the  complete  format  of  the 
^hierarchical  structure,  (2)  prior  probabilities  for  each 
^hypothesis  about  the  future  event,  (3)  probability  matrices 
for  the  conditioning  activities  and  indicators,  and  (4) 
direct  assessments  for  the  data  nodes.  Figure  6-2  shows  a 
representative  hierarchical  structure  for  a  hypothetical 
assessment  problem.  Mote  the  linkage  of  hypotheses,  sup¬ 
porting  activities,  key  indicators,  and  specific  data.  The 
figure  does  not  indicate  the  breakdown  of  the  nodes  into 
their  components  nor  does  it  show  the  specified  probability 
assessments. 

6.7  System  Outputs 

HIER  processes  the  user  input  specifications  to  produce 
the  posterior  probabilities  of  the  hypotheses  about  the 
uncertain  event.  Figure  6-3  shows  the  format  of  a  typical 
overall  result. 
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CURRENT  PROBABILITY  ASSESSMENTS: 


HYPOTHESES 

PRIORS 

LIKELIHOODS 

POSTERK 

1. 

Active  Cooperation 

25.00 

5.00 

6.25 

2. 

Reduced  Tensions 

20.00 

80.00 

80.00 

3. 

Political  Hostility 

20.00 

5.00 

5.00 

4. 

Active  Provocation 

30.00 

5.00 

7.50 

5. 

Aggression 

5.00 

5.00 

1.25 

Figure  6-3 
SAMPLE  RESULT 
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7 . 0  SCORING  RULE 


SCORING  RULE/  the  name  of  the  system  described  in  this 
•section/  is  a  short  description  of  the  function  performed  by 
the  software,  reflecting  the  system's  procedure  for  testing, 
scoring,  and  training  probability  assessors. 

7.1  Purpose 

The  overall  purpose  of  the  SCORING  RULE  system  is  to 
reduce  the  calibration  bias  of  individual  probability  asses¬ 
sors  and  thereby  improve  the  accuracy  of  probability  assess¬ 
ments.  That,  in  turn,  will  improve  the  accuracy  of  intelli¬ 
gence  estimates  and  the  quality  of  the  decisions  based  on 
those  estimates. 

7.2  Intended  Users 


Intelligence  analysts,  planners,  forecasters;  anyone 
who  must  analyze,  assess,  and  communicate  the  likelihood  of 
future  events. 

7.3  Potential  Area  of  Application 

o  Training  intelligence  analysts,  planners,  and 
forecasters 

7.4  Technical  Concepts  Underlying  the  Aid 

o  Subjective  probability 
o  Calibration  of  probability  assessments 

o  Proper  scoring  rules 
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7.5  System  Summary 

The  SCORING  RULE  system  administers  an  interactive 
subjective  probability  assessment  test  (SPAT)  to  the  user. 
€PAT  consists  of  a  series  of  short  questions,  each  of  which 
is  accompanied  by  two  answers,  only  one  of  which  is  correct. 
A  typical  question  appears  in  Figure  7-1. 


OF  THE  48  CONTIGUOUS  STATES,  THE  NORTHERNMOST  IS: 

1 .  MAINE 

2 .  MINNESOTA 


Figure  7-1 

A  TYPICAL  SPAT  QUESTION 


The  user  must  respond  to  each  question  by  identifying 
the  correct  answer  and  specifying  the  probability  (degree  of 
confidence)  that  the  cited  answer  is  correct.  The  system 
processes  the  user  responses  and  displays  the  results  in  the 
form  of  a  calibration  diagram,  as  shown  in  Figure  7-2.  The 
diagram  indicates  how  well  the  user's  assessments  corresponded 
with  the  actual  hit  rate  (the  percentage  of  questions  answered 
correctly) .  Ideally,  they  would  correspond  perfectly.  For 
example,  a  perfectly  calibrated  analyst  would  be  correct  in 
70%  of  all  estimates  that  were  assessed  as  70%  likely. 

7.6  System  Inputs 

The  input  to  the  SCORING  RULE  system  consists  of  re¬ 
sponses  to  the  SPAT  questions.  For  example,  a  typical 
response  to  the  question  asked  in  Figure  7-1  might  be: 

1  .8,  implying  that  the  user  is  80%  confident  that  Maine 
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ACTUAL  HIT  RATE 


O:  OPTIMAL  PERFORMANCE 
•:  ACTUAL  PERFORMANCE 


Figure  7-2 

CALIBRATION  DIAGRAM 


extends  farther  north  than  Minnesota.  Typically,  one  hun¬ 
dred  questions  and  responses  comprise  a  SPAT  session.  The 
questions  may  be  general  in  nature  or  related  to  a  special¬ 
ized  body  of  knowledge. 

7.7  System  Outputs 

At  the  user's  option,  SCORING  RULE  provides  question- 
by-question  feedback  that  indicates  whether  the  user's 
answer  is  right  or  wrong.  The  system  incorporates  a  scoring 
procedure,  known  theoretically  as  a  proper  scoring  rule, 
that  is  designed  to  reduce  guessing  by  assigning  a  win/lose 
point  score  consistent  with  the  user's  expressed  degree  of 
certainty.  Accordingly,  for  each  question  SCORING  RULE  also 
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displays  the  number  of  points  von  or  lost.  Thus,  the  sys¬ 
tem's  response  to  the  user's  answer  1  .8  to  the  question 

shown  in  Figure  7-1  would  be:  WRONG.  YOU  LOSE  49  POINTS. 

At  the  conclusion  of  the  test  session,  SCORING  RULE 
displays  the  resultant  calibration  diagram,  as  shown  in 
Figure  7-2,  and  an  overall  user  performance  measure,  ex¬ 
pressed  as  a  percentage  of  the  maximum  obtainable  calibra¬ 
tion  score. 


8.0  EVAL 


EVAL,  the  name  of  the  system  described  in  this  section, 
is  an  abbreviation  for  Evaluation,  reflecting  the  system's 
major  area  of  applicability. 

8.1  Purpose 

The  overall  purpose  of  EVAL  is  to  assist  decision 
makers  in  solving  problems  of  evaluation,  that  is,  in  dis¬ 
criminating  among  the  various  alternative  systems  or  other 
entities  or  courses  of  action  under  consideration.  The  EVAL 
approach  assumes  that  the  decision  maker  is  fully  informed 
about  the  alternatives  being  evaluated;  EVAL  does  not  treat 
uncertainty  explicitly. 

EVAL  ensures  that  the  final  evaluation  results  are 
totally  consistent  with  the  decision  maker's  expressed  goals 
and  value  structure.  Its  use  also  promotes  accountability 
in  the  evaluation  process  by  providing  an  audit  trail  that 
links  the  final  results  with  the  input  information. 

8 . 2  Intended  Users 


Decision  makers,  planners,  analysts,  operations  per¬ 
sonnel,  source  selection  staffs;  anyone  who  must  perform  an 
important  evaluation  and  selection  task. 

8.3  Potential  Areas  of  Application 

Procurement,  source  selection,  personnel  selection, 
comparative  systems  analysis,  planning,  any  type  of  formal 
evaluation  process. 


8 . 4  Technical  Concepts  Underlying  the  Aid 


o  Utility  assessment 
o  Multi-attribute  utility  theory 
o  Sensitivity  analysis 

8.5  System  Summary 

The  fundamental  product  of  EVAL  is  a  user-specified, 
computer- stored  evaluation  model.  The  model  accommodates 
multiple  evaluation  criteria  structured  in  hierarchical 
fashion,  as  shown  in  Figure  8-1. 

The  user  must  specify  the  hierarchical  format,  the 
criteria  and  their  relative  importance  weights,  and  values 
of  utility  for  each  of  the  entities  under  evaluation. 
Consistent  with  those  specifications,  EVAL  displays  the 
aggregate  utilities  with  respect  to  any  desired  criterion. 
The  overall  result  of  the  evaluation  is  a  display  of  com¬ 
parative  utilities  at  the  top-most  node  of  the  hierarchy. 
The  system  also  permits  the  user  to  perform  sensitivity 
analyses. 

8.6  User  Inputs 

The  user  must  specify  (1)  the  structural  format  of  the 
model,  (2)  the  names  of  the  criteria,  (3)  criterion  impor¬ 
tance  weights,  and  (4)  values  of  the  utility  of  each  entity 
under  evaluation  for  each  bottom-level  criterion  (those 
circled  in  Figure  8-1) . 

8.7  System  Outputs 

EVAL  processes  the  user  inputs  to  produce  comparative 
utilities  with  respect  to  any  specified  criterion  in  the 


TYPICAL  EVALUATION  MODEL 


modal.  Figure  8-2  shows  a  typical  output  display  £or  a  site 
selection  evaluation.  The  figure  shows  that  three  criteria 
comprised  the  overall  evaluation  result:  support  of  mission, 
support  of  relocation,  and  political  considerations.  The 
numbers  in  parentheses  indicate  the  relative  importance 
weights  of  the  criteria.  The  matrix  shows  the  aggregate 
utility  of  each  of  the  five  sites  being  evaluated  with 
respect  to  the  three  criteria.  The  result  of  interest  is 
the  total  utility  of  each  site,  which  is  shown  in  the  last 
row.  Consistent  with  the  input  information,  the  user 
should  select  Site  D,  which  provides  73%  of  complete  satis¬ 
faction  across  all  of  the  relevant  criteria. 


0.  OVERALL  RESULTS 


SITE 

SITE 

SITE 

SITE 

SITE 

A 

B 

C 

D 

E 

1. 

SUPPORT  OF  MISSION 

(52) 

61 

72 

45 

80 

75 

2. 

SUPPORT  OF  RELOCATION 

(33) 

20 

62 

70 

61 

38 

3. 

POLITICAL  CONSIDERA¬ 

TIONS 

(15) 

81 

43 

52 

76 

65 

TOTAL 

50 

64 

54 

73 

61 

Figure  8-2 

A  DISPLAY  OF  OVERALL  RESULTS 


EVAL  permits  the  user  to  vary  the  cumulative  importance 
weight  of  any  criterion  over  a  specified  range.  Figure  8-3 
shows  a  sample  sensitivity  analysis  in  which  the  weight  of 
particular  criterion  of  interest  was  varied  from  0  to  50%. 
The  figure  indicates  the  resultant  overall  utilities  of  the 
sites  under  evaluation  for  each  specified  weight.  An 
asterisk  identifies  the  site  having  the  greatest  utility. 
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Criterion  1.3. 3. 1.1 

SUPPORT  OF  MISSION— EXISTING  SPACE— AVAILABLE 
STORAGE— COVERED— QUALITY 


Current  CUMWTs  15 


SITE 

SITE 

SITE 

SITE 

SITE 

WEIGHT 

A 

B 

C 

D 

E 

0 

52 

61 

57 

*76 

58 

5 

51 

63 

56 

*75 

59 

10 

50 

64 

55 

*75 

60 

15 

50 

64 

54 

*73 

61 

20 

49 

66 

53 

*72 

62 

25 

48 

67 

52 

*70 

63 

30 

48 

69 

51 

*70 

65 

35 

47 

*71 

50 

68 

66 

40 

45 

*72 

50 

67 

66 

45 

45 

*74 

49 

65 

67 

50 

44 

*75 

48 

64 

68 

i 


Figure  8-3 

SAMPLE  SENSITIVITY  ANALYSIS 


RAM,  the  name  of  the  system  described  in  this  section, 
is  an  acronym  for  Resource  Allocation  Model,  reflecting  the 
system's  major  area  of  applicability. 

9.1  Purpose 

RAM  assists  decision  makers  by  prioritizing  the  order 
of  allocating  scarce  resources  to  acquire  competing  systems 
and/or  services.  Specifically,  RAM  was  designed  to  assist 
those  responsible  for  integrating,  preparing,  and  submitting 
the  annual  Program  Objectives  Memorandum  (POM)  within  the 
Department  of  Defense. 

9.2  Intended  Users 


Those  responsible  for  POM  preparation  and  submission, 
financial  officers,  and  others  who  must  allocate  scarce 
monetary  resources. 

9.3  Potential  Areas  of  Application 

o  POM  preparation 
o  Resource  allocation 

9.4  Technical  Concepts  Underlying  the  Aid 

o  Utility  assessment 
o  Cost-benefit  analysis 

9.5  System  Summary 


RAM  processes  cost  and  benefit  information  supplied  by 
the  user  to  produce  several  different  kinds  of  analyses  and 


reports  listing  ths  prioritised  order  of  acquisitions  and 
other  relevant  information.  RAM  permits  the  user  to  build, 
store,  retrieve,  revise,  and  exercise  different  resource 
allocation  models  and  to  display  and  print  a  variety  of 
different  reports  for  each  model. 

9.6  System  Inputs 

RAM  requires  that  the  user  specify  (1)  the  functional 
groupings  (such  as  logistics,  aviation,  etc.)  of  the  candi¬ 
date  acquisition  packages,  (2)  the  packages  belonging  to 
each  functional  grouping,  (3)  the  values  of  benefit  and  cost 
for  each  package  within  the  functional  grouping,  and  (4)  a 
relative  importance  weight  for  each  functional  grouping. 

9.7  System  Outputs 

RAM  processes  the  input  information  to  produce  various 
consolidated  reports.  For  example.  Figure  9-1  shows  a 
prioritized  list  of  acquisitions,  ordered  by  increasing 
cost-benefit  ratio.  Note,  however,  that  the  first  item  on 
the  list  had  been  declared  a  "must  buy"  item  by  the  user; 
therefore,  its  cost-benefit  ratio  was  not  considered  in  the 
prioritization  scheme.  The  numbers  to  the  left  of  each  item 
name  represent  the  functional  area  and  item  number,  respec¬ 
tively. 

Figure  9-2  shows  the  costs  and  the  cumulative  costs  of 
the  items  for  the  five  fiscal  years  from  1980  through  1984. 

Other  report  formats  are  available  to  the  user. 
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OVERALL  RAM 


ITEM 

SPONSOR 

BENEFIT 

OVERALL 

BENEFIT 

COST 

C/B 

RANK 

3 

2) FACULTY  RAISES 

30.0 

18.0 

1017.0 

56.6 

1 

4 

3) CAREER  PLACEMENT  CTR 

100.0 

35.0 

44.0 

1.3 

2 

1 

2) ALUMNI  SURVEY 

25.0 

2.5 

5.0 

2.0 

3 

2 

4 ) MAINT  &  RENOVATION 

100.0 

100.0 

250.0 

2.5 

4 

3 

1) INTERVIEW  &  HIRING 

55.0 

33.0 

100.0 

3.0 

5 

4 

5 ) EXTRACURRICULAR 

20.0 

7.0 

29.0 

4.1 

6 

2 

1)LAB  FACILITIES 

60.0 

60.0 

270.0 

4.5 

7 

S 

2) FEDERAL  RELATIONS 

90.0 

18.0 

89.0 

4.9 

8 

4 

1) STUDENT  CENTER 

30.0 

10.5 

52.0 

5.0 

9 

5 

1) ALUMNI  ASSOCIATION 

25.0 

5.0 

26.0 

5.2 

10 

4 

4) HOUSING  GUIDE 

15.0 

5.25 

31.0 

5.9 

11 

2 

5) INVESTMENT  AID 

12.0 

12.0 

72.0 

6.0 

12 

5 

3) FOUNDATION  SUPPORT 

100.0 

20.0 

132.0 

6.6 

13 

4 

2) COUNSELING  PROGRAM 

45.0 

15.75 

110.0 

7.0 

14 

3 

6) BOOKS  &  PERIODICALS 

100.0 

60.0 

425.0 

7.1 

15 

5 

5) RECRUITING 

70.0 

14.0 

103.0 

7.4 

16 

1 

3) ARCHITECTURAL  SURVEY 

100.0 

10.0 

80.0 

8.0 

17 

2 

3) IN-HOUSE  COMPUTER 

70.0 

70.0 

660.0 

9.4 

18 

4 

6) HEALTH  SERVICES 

75.0 

26.25 

278.0 

10.6 

19 

3 

5) FACULTY  OFFICE  BLDG 

80.0 

48.0 

810.0 

16.9 

20 

1 

4) EXPANS ION  PLANNING 

65.0 

6.5 

113.0 

17.4 

21 

5 

4) COMMUNITY  INVOLVEMENT 

50.0 

10.0 

244.0 

24.4 

22 

3 

4) BUSINESS  PROGRAM 

70.0 

42.0 

1200.0 

28.6 

23 

1 

1) SECRETARIAL  SERVICES 

10.0 

1.0 

95.0 

95.0 

24 

3 

3) NURSING  PROGRAM 

25.0 

15.0 

1870.0 

124.7 

25 

2 

2) ATHLETIC  COMPLEX 

5.0 

5.0 

5060.0 

1012.0 

26 

PLEASE  RETURN  CARRIAGE  TO  CONTINUE. 


Figure  9-1 

OVERALL  DISPLAY  OF  ITEMS  RANKED  IN  ASCENDING  ORDER 


Figure  9-2 
COST  OF  THE  ITEMS 


10.0  THE  DOCUMENTATION 


10.1  Background 

The  seven  decision  aids  described  herein  were  designed 
and  developed  as  research  products  rather  than  as  production 
software.  Because  of  their  research  nature ,  all  of  the  aids 
were  programmed  in  the  APL  computer  programming  language, 
which  was  ideally  suited  to  the  developmental  task.  For  the 
same  reason,  the  software  was  programmed  to  run  on  an  IBM  5100 
portable  computer.  In  that  particular  software  and  hardware 
configuration,  the  decision  aids  were  tested  and  applied  on 
a  pilot  basis  in  many  different  Department  of  Defense  (DoD) 
decision-making  contexts  and  locales  over  a  four-year  period. 
Many  of  those  applications  were  conducted  by  military  and 
civilian  DoD  personnel  who  had  received  specialized  class¬ 
room  and  on-the-job  training  on  the  theoretical  background 
and  practical  use  of  the  software.  In  other  applications 
skilled  contractor  decision  analysts  used  the  aids  to 
assist  in  the  analysis  and  presentation  of  substantive  in¬ 
formation  provided  by  DoD  experts.  In  all  instances,  the 
choice  of  the  APL  language  and  the  IBM  5100  computer  proved 
extremely  satisfactory  to  the  task  of  developing  and  experi¬ 
mentally  applying  the  decision  aids  to  real-world  decision 
problems. 

However,  neither  the  software  nor  the  hardware  is  suit¬ 
able  for  the  widespread  dissemination  and  adoption  of  the 
decision  aids  by  prospective  DoD  users.  The  reasons  for 
that  are  twofold:  first,  because  APL  is  not  an  approved 
standard  language  for  the  production  of  DoD  automated  sys¬ 
tems  (nor  is  APL  found  in  the  repertoire  of  most  DoD  pro¬ 
grammers)  and,  second,  because  the  IBM  5100  computer  is  not 
in  general  use  within  the  DoD  automated  data  processing 


community.  Furthermore,  it  would  prove  impractical  if  not 
impossible  to  cite  a  unique  software  and  hardware  configura¬ 
tion  upon  which  to  base  a  documentation  effort  that  would 
satisfy  all  of  the  prospective  users  of  the  aids. 

Accordingly,  to  ensure  its  most  widespread  applica¬ 
bility,  the  documentation  described  herein  is  fundamentally 
generic  in  nature.  That  is,  all  of  the  documents,  with  only 
one  exception  (the  HIER  System  Specification) ,  are  written 
without  reference  to  any  specific  programming  language  or 
computer  configuration.  The  documents  have  been  designed  to 
enable  responsible  personnel  at  prospective  user  activities 
to  produce  the  language-specific  computer  programs  and 
hardware-specific  documentation  necessary  to  implement  the 
decision  aids  on  their  own  particular  computer  facilities. 
The  generic  documents  described  herein  should  greatly  ease 
the  prospective  user's  task  of  preparing  the  system-specific 
documents  necessary  to  implement  the  aids. 

10.2  Types  of  Documents 

In  accordance  with  accepted  DoD  standards,  a  complete 
software  documentation  effort  should  address  five  different 
audiences:  (1)  the  user  of  the  software  system;  (2)  the 

functional  manager  of  the  system;  (3)  the  computer  systems 
analyst;  (4)  the  computer  programmer;  and  (5)  the  computer 
operator.  Each  audience  has  different  needs  and  requires  a 
specialized  document  designed  to  support  those  needs. 

Consider  the  audience  at  a  hypothetical  activity  that 
intends  to  implement  one  of  the  aids.  First,  there  is  the 
user— that  person  who  wants  to  use  the  aid  in  a  real-world 
context.  An  intelligence  analyst,  for  example,  may  want  to 
use  the  INFER  aid.  The  user  requires  a  Users  Manual  that 
explains  the  aid  and  describes  how  to  interface  with  the 
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computer  to  use  the  aid.  The  Users  Manual  is  written  in 
nontechnical  language  relative  to  computer  science,  although 
it  may  use  technical  language  relative  to  the  application  of 
the  aid. 

Second,  there  is  the  functional  manager  of  the  software 
system'— that  person  who  is  responsible  for  the  implementation 
and  performance  of  the  aid.  For  the  INFER  aid,  for  example, 
the  functional  manager  may  be  the  Director  of  the  Intelligence 
Staff.  The  functional  manager  originates,  maintains,  and 
distributes  the  Users  Manual  as  well  as  a  Functional  Descrip¬ 
tion  document  that  lists  and  describes,  in  nontechnical 
language  relative  to  computer  science,  the  specific  functions 
that  the  aid  must  perform.  The  Functional  Description  ex¬ 
plains  to  computer  systems  development  personnel  what  the 
system  must  do. 

The  third  audience,  the  technical  computer  systems 
analyst,  uses  the  Functional  Description  to  design  the  logic 
of  the  software  system  and  produce  a  formal  System  Specifi¬ 
cation  that  serves  as  the  basis  of  the  software  production 
task.  Thus,  the  System  Specification  is  a  technical  docu¬ 
ment  relative  to  computer  science,  and  is  normally  based  on 
a  specific  software  and  hardware  configuration.  The  computer 
systems  analyst  also  assists  the  functional  manager  in  pre¬ 
paring  the  computer  interface  portion  of  the  Users  Manual. 

Fourth,  there  is  the  computer  programmer,  who  uses  the 
System  Specification  to  prepare  a  Program  Specification  docu¬ 
ment  and  language-specific  program  code.  The  programmer 
also  prepares  a  Program  Maintenance  Manual,  which  is  con¬ 
sulted  by  the  fifth  audience,  the  technical  personnel  who 
operate  and  maintain  the  computer  facility. 
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Figure  10-1  illustrates  the  five  levels  of  audience 
interaction  with  a  software  system  and  the  necessary  sup¬ 
porting  documents:  the  Users  Manual,  Functional  Descrip¬ 
tion,  System  Specification,  Program  Specification,  and 
Program  Maintenance  Manual.^ 

10.3  Scope  of  the  Documentation  Effort 

To  satisfy  their  intended  audiences,  some  of  the  docu¬ 
ments  shown  in  Figure  10-1  are  computer  software-  and  hard¬ 
ware-specific  and  others  are  not.  For  example,  the  Program 
Specification  and  Program  Maintenance  Manual  are  completely 
software-  and  hardware-specific.  They  could  not  be  included 
in  this  generic  documentation  effort.  On  the  other  hand, 
the  Functional  Description  does  not  reference  computer  soft¬ 
ware  or  hardware — it  need  only  address  the  functions  that 
the  decision  aid  must  perform?  it  is  generic  by  its  very 
nature . 

The  Users  Manual  and  the  System  Specification  ordi¬ 
narily  reference  the  hardware  configuration.  The  user,  for 
example,  must  be  told  how  to  interface  with  the  computer. 
Nevertheless,  both  documents  can  be  written  independently  of 
language  and  computer. 

Accordingly,  to  preserve  its  generic  flavor,  the  scope 
of  the  documentation  of  the  decision-aiding  software  has 
been  restricted  to  three  documents  for  each  aid.  The  docu¬ 
ments  comprise  a  Users  Manual,  a  Functional  Description,  and 
a  System  Specification. 


T - 

The  five  documents  are  necessary  but  not  necessarily  suf- 
fucient  to  completely  document  a  software  system. 
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LEVELS  Or 
INTERACTION 


DOCUMENTATION 


f' 


THE 

FUNCTIONAL 

MANAGER 


preg*r£' 


USERS 

MANUAL 


FUNCTIONAL 


SYSTEMS 

ANALYST 


COMPUTER 

PROGRAMMER 


SPECIFICATION 


\ 

\ 

^  | 

SPECIFICATION 

\ 

\ 

COMPUTER 

OPERATOR 

(/SJS 

PROGRAM 

MAINTENANCE 

MANUAL 

Figure  10-1 

LEVELS  or 

DOCUMENTATION  AND  USERS  OF  DOCUMENTATION 

The  three  documents  are  intended  to  guide  and  facili¬ 
tate  the  preparation  of  language-specific  and  hardware- 
specific  program  code  and  documentation  by  a  using  activity. 
All  three  documents  are  necessary  to  support  a  follow-on 
software  development  task  at  a  using  activity.  The  purpose 
of  each  document  is  described  below. 

10.3.1  The  Users  Manual  -  The  purpose  of  the  Users 
Manual  is  to  provide  users  of  the  decision  aid  with  the 
background  material  and  the  detailed  instructions  necessary 
to  interface  with  the  aid  and  interpret  the  various  func¬ 
tions  that  the  aid  provides.  The  manual  presents  the 
decision-analytic  concepts  inherent  in  the  aid,  including 
assumptions  and  restrictions  concerning  the  use  of  the  aid. 
Most  of  the  manuals  include  descriptive  case  studies. 

10.3.2  The  Functional  Description  -  The  Functional 
Description  provides  a  delineation  of  the  specific  functions 
that  the  software  must  perform.  It  serves  as  a  formal  basis 
for  understanding  between  the  functional  manager  of  the  aid 
and  the  software  development  personnel.  Together  with  the 
System  Specification,  it  serves  as  the  basic  reference 
documentation  for  the  software  development  and  implementa¬ 
tion  task. 

10.3.3  The  System  Specification  -  The  System  Speci¬ 
fication  is  a  technical  document  written  exclusively  for 
software  development  personnel.  Together  with  the  Func¬ 
tional  Description,  it  guides  the  software  development 
effort  by  identifying  the  functional  requirements  of  the 
software  system  and  by  providing  structured  logic  diagrams 
that  depict  the  flow,  control,  and  processing  of  information 
within  the  system.  For  six  of  the  aids,  the  System  Specifi¬ 
cation  uses  a  standard  hierarchical  diagramming  technique  to 
depict  the  structural  design  and  the  logical  flow  of  the 


•y»tem.  For  the  seventh  (the  HIER  aid)  the  System  Specifi¬ 
cation  consists  of  APL  programming  code. 

10.4  List  of  Documents 

The  following  is  a  list  of  the  twenty-one  manuals  that 
document  the  seven  aids. 

10.4.1  The  DECISION  aid  - 

o  Allardyce,  Linda  B.j  Amey,  Dorothy  M.;  Feuerwerger, 
Phillip  H.;  and  Gulick,  Roy  M.  Documentation 
of  Decision-Aiding  Software:  DECISION  Users 
Manual .  McLean ,  Virginia:  Decisions  and 
Designs ,  Inc.,  November  1979. 

o  Allardyce,  Linda  B.;  Amey,  Dorothy  M.;  Feuerwerger, 
Phillip  H. ;  and  Gulick,  Roy  M.  Documentation 
of  Decision- Aiding  Software:  DECISION  Func¬ 
tional  Description.  McLean,  Virginia:  Deci¬ 
sions  and  Designs,  Inc.,  November  1979. 

o  Allardyce,  Linda  B. ;  Amey,  Dorothy  M .;  Feuerwerger, 
Phillip  H. ?  and  Gulick,  Roy  M.  Documentation 
of  Decision-Aiding  Software:  DECISION  System 
Specification.  McLean,  Virginia:  Decisions 
and  Designs,  Inc.,  November  1979. 

10.4.2  The  OPINT  aid  - 

o  Amey,  Dorothy  M.j  Feuerwerger,  Phillip  H.;  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  OPINT  Users  Manual.  McLean, 
Virginia:  Decisions  and  Designs,  Inc.,  April 
1979. 
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o  Amey,  Dorothy  M. »  Feuerwerger,  Phillip  H. r  and 
Gulick,  Soy  M.  Documentation  of  Decision- 
Aiding  Softwara:  OPINT  Functional  Description. 
McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  April  1979. 

o  Amey,  Dorothy  M.i  Feuerwerger,  Phillip  H.:  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  OPINT  System  Specification. 
McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  April  1979. 

10.4.3  The  INFER  aid  - 

o  Amey,  Dorothy  M.;  Feuezverger,  Phillip  H.:  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  INFER  Users  Manual.  McLean, 
Virginia:  Decisions  and  Designs,  Inc.,  June 
1979. 

o  Amey,  Dorothy  M. ;  Feuerwerger,  Phillip  H. :  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  INFER  Functional  Description. 
McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  June  1979. 

o  Amey,  Dorothy  M. >  Feuerwerger,  Phillip  H.;  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  INFER  System  Specification. 
McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  June  1979. 
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10.4.4  The  HIER  aid 


o  Amey,  Dorothy  M. t  Gulick,  Roy  M. t  end  Kelly, 

Clinton  w.  Documentation  of  Dec! sion- Aiding 
Software:  HIER  Ueers  Manual.  McLean,  Vir¬ 
ginia:  Decisions  and  Designs,  Inc.,  in 
press. 

o  Amey,  Dorothy  M.;  Gulick,  Roy  M .;  and  Kelly, 

Clinton  W.  Documentation  of  Decision-Aiding 
Software:  HIER  Functional  Description. 

McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  in  press. 

o  Amey,  Dorothy  M.;  Gulick,  Roy  M. ;  and  Kelly, 

Clinton  W.  Documentation  of  Dec is ion- Aiding 
Software:  HIER  System  Specification.  McLean , 
Virginia:  Decisions  and  Designs,  Inc.,  in 
press. 

10.4.5  The  SCORING  RULE  aid  - 

o  Amey,  Dorothy  M .;  Feuerwerger,  Phillip  H.;  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  SCORING  RULE  Users  Manual. 
McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  July  1979. 

o  Amey,  Dorothy  M. ;  Feuerwerger,  Phillip  H. ;  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  SCORING  RULE  Functional 
Description.  McLean,  Virginia:  Decisions 
and  Designs,  Inc.,  July  1979. 


o  Amey,  Dorothy  M. j  Feuerwerger ,  Phillip  H. ;  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  SCORING  RULE  System  Speci¬ 
fication.  McLean,  Virginia:  Decisions  and 
Designs,  Inc.,  July  1979. 

10.4.6  The  BVAL  aid  - 

o  Allardyce,  Linda  B. ;  Amey,  Dorothy  M. ;  Feuerwerger, 
Phillip  H.;  and  Gulick,  Roy  M.  Documentation 
of  Decision-Aiding  Software:  EVAL  Users  Manual. 
McLean,  Virginia:  Decisions  and  Designs, 

Inc.,  November  1979. 

o  Allardyce,  Linda  B.?  Amey,  Dorothy  M. ;  Feuerwerger, 
Phillip  H.;  and  Gulick,  Roy  M.  Documentation 
of  Decision- Aiding  Software:  EVAL  Functional 
Description.  McLean,  Virginia:  Decisions 
and  Designs,  Inc.,  November  1979. 

o  Allardyce,  Linda  B. j  Amey,  Dorothy  M. ;  Feuerwerger, 
Phillip  H.;  and  Gulick,  Roy  M.  Documentation 
of  Decision-Aiding  Software:  EVAL  System  Spe¬ 
cification.  McLean,  Virginia:  Decisions  and 
Designs,  Inc.,  November  1979. 

10.4.7  The  RAM  aid  - 

o  Amey,  Dorothy  M. ;  Feuerwerger,  Phillip  H. :  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  RAM  Users  Manual.  McLean , 
Virginia:  Decisions  and  Designs,  Inc., 
September  1979. 
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Amey,  Dorothy  M. ;  Feuerwerger,  Phillip  H.i  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  RAM  Functional  Description. 
McLean,  Virginia:  Decisions  and  Designs, 
Inc.,  September  1979. 

Amey,  Dorothy  M. ;  Feuerwerger,  Phillip  H.;  and 
Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  RAM  System  Specification. 
McLean,  Virginia:  Decisions  and  Designs, 
Inc.,  September  1979. 


